REPORT  DOCUMENTATION  PAGE 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  this  coilection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the 
data  needed,  and  completing  and  reviewing  this  coilection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  coliection  of  information,  including  suggestions  for  reducing 
this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202- 
4302.  Respondents  shouid  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently 
valid  0MB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1.  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE 

30-01-2015  Journal  Article 

3.  DATES  COVERED  (From  -  To) 

IMay  2014-30  Sep  2014 

4.  TITLE  AND  SUBTITLE 

Predictive  Validity  of  UAS/RPA  Sensor  Operator  Training  Qualification  Measures. 

5a.  CONTRACT  NUNA 

6.  AUTHOR(S) 

Carretta,  Thomas  R. 

Rose,  Mark  R. 

Barron,  Laura  G. 

5d.  PROJECT  NUMBER 

5329 

5e.  TASK  NUMBER 

09 

5f.  WORK  UNIT  NUMBER 

H03K/53290902 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES)  AND  ADDRESS 

See  next  page. 

8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 

N/A 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Materiel  Command 

Air  Force  Research  Laboratory 

711  Human  Performance  Wing 

Human  Effectiveness  Directorate 

Warfighter  Interface  Division 

Supervisory  Control  and  Cognition  Branch 

Wright-Patterson  AFB  OH  45433 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

711  HPW/RHCI 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

N/A 

12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

Distribution  A:  Approved  for  public  release;  distribution  unlimited. 

13.  SUPPLEMENTARY  NOTES 

Published  in  the  International  Journal  of  Aviation  Psychology,  Volume  25,  Issue  I,  pages  3-13. 

88ABW  Cleared  03/02/2015;  88BW-2015-0759. 

14.  ABSTRACT 

The  jobs  of  unmanned  aerial  system/remotely  piloted  aircraft  (UAS/RPA)  pilot  and  sensor  operator  (SO)  are  relatively  new  in  aviation  and 
views  on  who  is  best  suited  to  fill  these  positions  are  evolving.  There  are  few  published  studies  that  have  examined  the  entry  requirements 
for  UAS/RPA  training  or  the  predictive  utility  of  selection  methods.  In  the  US  Air  Force  (USAF)  selection  methods  and  aptitude 
requirements  for  UAS/RPA  pilots  are  very  similar  to  those  for  manned  aircraft  pilots  and  have  shown  similar  levels  of  predictive  validity 
for  both.  USAF  SOs  are  enlisted  aviators  who  work  side-by-side  with  UAS/RPA  pilots,  providing  assistance  with  all  aspects  of  aircraft 
employment  and  sensor  management.  Qualification  for  SO  training  involves  medical,  citizenship,  and  security  standards  and  aptitude 
requirements  (Armed  Services  Vocational  Aptitude  Battery  [ASVAB]).  The  current  study  examined  the  predictive  validity  of  the  ASVAB 
for  SO  training  grades.  Participants  were  students  enrolled  in  three  SO  courses  (Basic  Sensor  Operator  Course  [BSOC]  n  =  461,  MQ-1 

Initial  Qualification  Requirements  Training  [MQ-1  IQRT]  n  =  430,  or  MQ-9  Initial  Qualification  Training  MQ-9  [IQRT],  n  =  249).  The 
training  criterion  was  average  grade  on  written  tests.  The  two  ASVAB  composites  used  for  SO  training  qualification  (General  and 
Electronics)  demonstrated  good  predictive  validity  for  all  three  courses  (BSOC,  .541  and  .535;  MQ-1  IQRT,  .583  and  .553;  MQ-9  IQRT, 
(continued  on  next  page) 

15.  SUBJECT  TERMS 

Sensor  operators.  Armed  Services  Vocational  Aptitude  Battery,  ASVAB 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION 

OF  ABSTRACT 

SAR 

18.  NUMBER 
OF  PAGES 

23 

19a.  NAME  OF  RESPONSIBLE  PERSON 

Thomas  R.  Carretta 

a.  REPORT 

Unclassified 

b.  ABSTRACT 

Unclassified 

c.  THIS  PAGE 

Unclassified 

19b.  TELEPHONE  NUMBER  (include  area 
code) 

standard  Form  298  (Rev.  8-98) 


Prescribed  by  ANSI  Std.  Z39.18 


7.  Performing  Organizations 


Thomas  R.  Carretta 
Air  Force  Research  Laboratory 
711  HPW/RHCI 
2210  8*  Street 

Area  B,  Bldg.  146,  Room  122 
Wright-Patterson  AFB,  OH  45433-7511 

Mark  R.  Rose 

Laura  G.  Barron 

Air  Force  Personnel  Center 

HQ  AFPC/DSYX 

550  C  Street  W,  Ste.  45 

Randolph  AFB,  TX  78150-4747 

14.  Abstract  (cont.) 

.357  and  .334).  Although  current  selection  methods  are  effective,  based  on  results  of  UAS/RPA  job/task  analyses,  the  Air  Force 
examining  the  utility  of  other  measures  to  supplement  the  ASVAB. 


UAS/RPA  SENSOR  OPERATOR  TRAINING  QUALIFICATION  1 


Predictive  Validity  of  UAS/RPA  Sensor  Operator  Training  Qualification  Measures 

Thomas  R.  Carretta 
Air  Force  Research  Laboratory 
Wright-Patterson  AFB,  OH 

Mark  R.  Rose 
Laura  G.  Barron 
Air  Force  Personnel  Center 
Randolph  AFB,  TX 


The  opinions  expressed  are  those  of  the  authors  and  not  necessarily  those  of  the  United  States 
Government,  Department  of  Defense,  or  the  United  States  Air  Force.  Address  all  correspondence 
to  Dr.  Thomas  R.  Carretta,  711  HPW/RHCI,  2210  8*  Street,  Area  B,  Bldg.  146,  Room  122, 
Wright-Patterson  AFB,  OH  45433-7511.  Send  e-mail  to  thomas .carretta@us . af.mil. 


Distribution  A:  Approved  for  public  release;  distribution  unlimited. 
88ABW  Cleared  03/02/2015;  88ABW-2015-0759. 

International  Journal  of  Aviation  Psychology,  Volume  25,  Issue  1,  pages  3-13 


UAS/RPA  SENSOR  OPERATOR  TRAINING  QUALIFICATION  2 
Abstract 

US  Air  Force  sensor  operators  (SOs)  are  enlisted  aviators  who  work  side-by-side  with  unmanned 
aerial  systems/remotely  piloted  aireraft  (UAS/RPA)  pilots,  providing  assistanee  with  all  aspeets 
of  aireraft  employment  and  sensor  management.  SO  training  qualification  includes  medieal, 
eitizenship,  and  seeurity  standards  and  aptitude  requirements.  The  eurrent  study  examined  the 
validity  of  the  Armed  Serviees  Voeational  Aptitude  Battery  (ASVAB)  for  predieting  grades  of 
students  in  three  SO  eourses.  The  ASVAB  eomposites  used  for  SO  training  qualifieation 
(General  and  Electronies)  demonstrated  good  predietive  validity  for  all  three  eourses  (eorrected 
for  range  restrietion  and  eriterion  unreliability):  Basie  Sensor  Operator  Course,  n  =  461,  r  =  .541 
and  .535;  MQ-1  Initial  Qualification/Requalifieation  Training,  n  =  430,  r  =  .583  and  .553;  MQ-9 
Initial  Qualifieation/Requalifieation  Training,  n  =  249,  r  =  .357  and  .334).  Although  eurrent 
selection  methods  are  effective,  based  on  results  of  UAS/RPA  job/task  analyses,  the  Air  Force  is 
examining  the  utility  of  other  measures  to  supplement  the  ASVAB. 

Key  Words:  unmanned  aerial  systems,  remotely  piloted  aireraft,  sensor  operators,  personnel 
seleetion.  Armed  Serviees  Voeational  Aptitude  Battery 
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Predictive  Validity  of  UAS/RPA  Sensor  Operator  Training  Qualification  Measures 

The  jobs  of  unmanned  aerial  system/remotely  piloted  aireraft  (UAS/RPA)  pilot  and 
sensor  operator  are  relatively  new  in  aviation  and  views  on  who  is  best  suited  to  fill  these 
positions  are  evolving  (Carretta,  Rose,  &  Bruskiewicz,  in  press).  The  US  military  employs 
several  UAS/RPA  systems  whieh  vary  in  size,  configuration,  and  characteristics.  There  are  few 
published  studies  that  have  examined  the  entry  requirements  for  UAS/RPA  training  or  the 
predictive  utility  of  selection  methods  (for  a  review  see  Carretta  et  ah,  in  press). 

US  Air  Force  (USAF)  RPA  Pilots 

USAF  RPA  pilots  are  a  mix  of  1)  rated  officers  trained  to  fly  medium- altitude, 
weaponized  aircraft  (HQ  AFPC/DPSIDC,  2011a)  who  have  been  retrained  to  operate  RPAs  and 
2)  officers  with  no  prior  operational  flying  experience  who  have  completed  the  Undergraduate 
RPA  Training  (URT)  course.  Currently,  the  majority  of  new  USAF  RPA  pilots  are  URT 
graduates  with  the  remainder  coming  from  retrained  experienced  pilots. 

Selection  methods  and  aptitude  requirements  for  URT  are  very  similar  to  those  for 
manned  aircraft  pilot  training  (Carretta,  2013;  Rose,  Arnold,  &  Howse,  2013).  Although 
selection  factors  vary  by  officer  commissioning  source,  medical  flight  screening  and  aptitude 
testing  are  two  important  factors.  Medical  Flight  Screening  (MFS)  includes  successful 
completion  of  an  FAA  Class  III  Medical  Certificate  and  an  USAF  Flying  Class  IIU  Medical 
Examination  (United  States  Air  Force,  2011),  review  of  medical  records,  psychological  testing, 
and  an  interview.  A  licensed  psychologist  uses  clinical  judgment  to  assess  the  psychological 
disposition  of  URT  applicants  to  determine  whether  there  is  a  medically  disqualifying  condition 
in  compliance  with  Air  Force  guidelines  (United  States  Air  Force,  2011).  Aptitude  testing 
includes  the  Air  Force  Officer  Qualifying  Test  (AFOQT;  Drasgow,  Nye,  Carretta,  &  Ree,  2011), 
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Test  of  Basic  Aviation  Skills  (TEAS;  Carretta,  2005),  and  Pilot  Candidate  Selection  Method 
(PCSM;  Carretta,  2011).  Minimum  qualifying  scores  for  URT  have  been  established  for  both  the 
AFOQT  and  PCSM  and  are  identical  to  those  for  manned  aircraft  pilot  training  (United  States 
Air  Force,  2014). 

US  Air  Force  -  Sensor  Operators 

In  2009,  the  USAF  also  created  the  UAS  Sensor  Operator  (SO)  career  field  (lUOXl). 
Sensor  operators  are  enlisted  aviators  who  work  side-by-side  with  RPA  pilots,  providing 
assistance  to  the  pilot  with  all  aspects  of  aircraft  employment  and  sensor  systems  management 
(HQ  AFPC/DPSIDC,  2011b).  They  are  responsible  for  the  employment  of  airborne  sensors  in 
manual  or  computer-assisted  modes  to  acquire,  track,  and  monitor  airborne,  ground,  and 
maritime  objects.  Sensor  operators  assist  RPA  pilots  through  all  phases  of  employment 
including  mission  planning,  flight  operations,  and  debriefings.  They  continually  monitor  aircraft 
and  weapons  systems  status  to  ensure  lethal  and  non-lethal  application  of  airpower.  Currently, 
USAF  lUOXl  specialists  perform  their  duties  on  the  MQ-1  Predator  and  the  MQ-9  Reaper 
RPAs.  Selection  for  enlisted  SOs  is  based  on  medical  qualification  (United  States  Air  Force, 
2011),  US  citizenship,  eligibility  for  a  Top  Secret  clearance,  and  scores  on  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB;  Segall,  2004),  the  US  military  enlistment  qualification 
and  training  classification  test.  Courses  in  chemistry,  computer  science,  earth  sciences, 
geography,  and  math  are  desirable. 

Job/Task  Analyses 

Several  studies  have  examined  UAS/RPA  job  requirements  and  the  critical  skills, 
abilities,  and  other  characteristics  (SAOCs)  needed  to  perform  them.  In  1979,  the  US  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  conducted  individual  structured 
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interviews  with  UAS  test  system  personnel  (operators,  support  personnel,  and  supervisors)  as 
part  of  a  research  program  to  establish  selection  criteria  for  UAS  personnel  (Crumley  &  Bailey, 
1979).  These  data  were  used  to  guide  structured  group  interviews  with  UAS  operators. 

Results  showed  that  both  the  air  vehicle  operator  (AVO)  and  sensor  operator  (SO)  jobs 
required  average  or  above-average  hand/eye  coordination,  verbal  communication  skill,  logic,  and 
patience,  but  lower  levels  of  color  vision,  endurance,  hearing  acuity,  and  physical  strength. 
Further,  AVOs  and  SOs  were  found  to  have  distinct  personality  characteristics.  AVOs  tended  to 
like  the  logic,  planning,  and  sequential  steps  of  the  position,  whereas  sensor  operators  tended  to 
prefer  the  excitement,  uncertainty,  and  unpredictability  associated  with  the  task.  Finally,  both 
AVOs  and  SOs  expressed  dislike  for  close  supervision  during  missions,  but  nevertheless  favored 
having  a  third  (artillery  trained)  person  in  the  Ground  Control  Station  (Crumley  &  Bailey,  1979). 

US  Air  Force  and  UK  Royal  Air  Force  studies  also  have  found  overlap  in  the  entry-level 
SAOCs  required  for  RPA  pilots  and  SOs  (e.g.,  see  Carretta  et  ah,  in  press  or  Howse,  2011  for  a 
review).  For  example,  results  of  job  analyses  have  shown  central  information  processing  (e.g., 
timesharing),  perceptual  and  psychomotor  abilities,  spatial  and  symbolic  reasoning,  situational 
awareness,  and  working  memory  to  be  essential  for  RPA  pilot  and  sensor  operator  performance 
(Bailey,  2008;  Chappelle,  McDonald,  &  King,  2010;  Chappelle,  McDonald,  &  McMillan,  2011; 
Paullin,  Ingerick,  Trippe,  &  Wasko,  2011). 

Predictive  Validation  Studies 

Three  recent  studies  have  examined  the  predictive  validity  of  aptitude  and  personality 
measures  forURT  (Carretta,  2013;  Chappelle,  McDonald,  Heaton,  Thompson,  &  Haynes,  2012; 
Rose,  Barron,  Carretta,  Arnold,  &  Howse,  2014).  Carretta  (2013)  examined  the  predictive 
validity  of  the  AFOQT  Pilot  and  PCSM  composites  for  URT  completion  (pass/fail)  for  a  sample 
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of  139  students.  Moderate  validities  for  the  AFOQT  Pilot  (r  =  .378,  p  <  .001)  and  PCSM  (r  = 
.480,  p  <  .001)  eomposites  were  observed  for  URT  completion.  The  correlations  increased  after 
correction  for  direct  range  restriction  (Thorndike,  1949)  and  dichotomization  (Cohen,  1983)  of 
the  URT  pass/fail  criterion  (AFOQT  Pilot,  r  =  .57;  PCSM,  r  =  .68).  Students  with  PCSM  scores 
at  or  below  the  25**^  percentile  had  a  52.2%  failure  rate.  Those  with  PCSM  scores  at  or  above  the 
75*  percentile  had  only  a  7.8%  failure  rate. 

Chappelle  et  al.  (2012)  and  Rose  et  al.  (2014)  explored  the  utility  of  using  a  combination 
of  cognitive  and  personality  scores  to  predict  URT  performance.  In  a  sample  of  1 17  URT 
students,  Chappelle  et  al.  examined  the  predictive  validity  of  the  AFOQT  Pilot  composite. 
Revised  NEO  Personality  Inventory  (NEO-PI-R;  Costa  &  McCrae,  1992),  and  a 
neuropsychological  battery,  the  MicroCog  (Powell,  Kaplan,  Whitla,  Weintraub,  Catlin,  & 
Funkenstein,  2004)  versus  URT  completion.  The  regression- weighted  composite  that  best 
predicted  URT  completion  included  the  AFOQT  Pilot  composite,  several  NEO-PI-R  scales,  and 
the  MicroCog  Reaction  Time  subtest.  Discriminant  analyses  showed  that  personality  scores 
improved  classification  accuracy  (identification  of  true  positives  and  true  negatives)  beyond  that 
provided  by  cognitive  ability  and  prior  flight  time.  When  personality  scores  were  included 
classification  accuracy  improved  from  57.1%  to  75.2%;  however,  these  results  likely  capitalized 
on  chance  given  the  large  number  of  NEO-PI-R  scales  relative  to  the  small  sample  size. 

Rose  et  al.  (2014)  evaluated  the  extent  to  which  scores  from  a  Big  Five  measure  of 
personality,  the  Self-Description  Inventory+  (SDI+;  Manley,  2011)  could  improve  prediction  of 
URT  outcomes  beyond  updated  versions  of  the  AFOQT  Pilot  and  PCSM  composites.  Participants 
were  170  URT  students  with  RPA  Initial  Flight  Screening  (RFS)  outcomes  and  1 10  students  with 
RPA  Instrument  Qualification  (RIQ)  outcomes.  Both  RFS  and  RIQ  emphasize  development  of 
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skills  traditionally  required  for  manned  aircraft  pilots,  also  considered  essential  for  USAF  RPA 
pilots.  Training  criteria  were  RES  completion  (pass/fail)  and  several  measures  involving  RIQ 
academic  and  simulator  grades.  Data  for  all  analyses  were  corrected  for  direct  range  restriction 
(Thorndike,  1949).  Data  for  analyses  involving  the  RES  completion  (pass/fail)  criterion  were 
corrected  both  for  direct  range  restriction  and  dichotomization  (Cohen,  1983).  Results  for  the  RES 
completion  criterion  were  similar  to  those  reported  by  Carretta  (2013).  Correlations  between  the 
updated  AFOQT  and  PCSM  composites  and  RES  completion  were  statistically  significant 
(AFOQT  Pilot,  r  =  .38,  p  <  .001;  PCSM,  r  =  .48,  p  <  .001).  Correlations  increased  after  correction 
for  direct  range  restriction  and  dichotomization  of  the  RES  pass/fail  criterion  (AFOQT  Pilot,  r  = 
48;  PCSM,  r  =  .67).  Analyses  involving  personality  scores  from  the  SDI+  showed  no 
incremental  validity  when  used  in  combination  with  the  AFOQT  Pilot  or  PCSM  composites  for 
predicting  RES  completion.  However,  the  SDI+  Openness  score  demonstrated  small,  but 
statistically  significant  incremental  validity  for  predicting  overall  RIQ  course  grades  aggregated 
across  academic  and  simulated  flying  performance. 

Purpose 

To  date,  no  studies  have  been  reported  that  examine  the  predictive  validity  of  selection 
methods  for  USAF  UAS/RPA  sensor  operators.  The  purpose  of  the  current  study  is  to  examine 
the  predictive  utility  of  cognitive  ability  and  technical  knowledge  scores  for  three  sensor  operator 
training  courses. 


Participants 


Method 
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Three  separate  validation  studies  were  condueted.  The  samples  consisted  of  enlisted 
personnel  enrolled  in  the  Basic  Sensor  Operator  Course  (BSOC,  n  =  461),  MQ-IB  Initial 
Qualification  and  Requalification  Training  course  (MQ-1  IQRT,  n  =  430),  or  the  MQ-9  Initial 
Qualification  and  Requalification  Training  course  (MQ-9  IQRT,  n  =  249).  To  qualify  for 
entrance  into  the  US  Air  Force,  applicants  must  achieve  a  score  at  or  above  the  36*  percentile  on 
the  ASVAB  AFQT  composited  Aptitude  requirements  for  sensor  operator  (lUOXl)  training  are 
a  score  at  or  above  the  64*  percentile  on  the  USAF  ASVAB  General  composite  or  a  score  at  or 
above  the  54*  percentile  on  the  Electronics  composite^.  Demographic  information  was  not 
available  for  all  participants.  However,  for  those  reporting  demographic  data,  the  results  were 
similar  across  the  three  samples.  The  participants  were  predominantly  male  (84.5  %  to  90.2%), 
White  (80.0%  to  90.1%),  and  non-Hispanic  (79.8%  to  82.4%).  The  mean  ages  for  the  samples 
ranged  from  19.1  to  20.7  years. 

Measures 

Armed  Services  Vocational  Aptitude  Battery  (ASVAB),  The  ASVAB  (Segall,  2004) 
is  used  for  enlistment  qualification  and  classification  into  training  specialties  by  all  branches  of 
the  US  military.  It  consists  of  9  tests:  General  Science  (GS),  Arithmetic  Reasoning  (AR),  Word 
Knowledge  (WK),  Paragraph  Comprehension  (PC),  Auto  and  Shop  Information  (AS), 
Mathematics  Knowledge  (MK),  Mechanical  Comprehension  (MC),  Electronics  Information  (El), 
and  Assembling  Objects  (AO).  The  two  verbal  (WK,  PC)  and  two  math  (AR,  MK)  tests 
contribute  to  the  Armed  Forces  Qualification  Test  (AFQT)  composite,  which  is  used  by  all 


*  The  AFQT  (Armed  Forces  Qualification  Test)  is  a  weighted  composite  of  the  ASVAB  Verbal  (VE)  composite 
(Paragraph  Comprehension  and  Word  Knowledge  tests)  and  math  tests  (Arithmetic  Reasoning  and  Math 
Knowledge). 

^  The  USAF  General  composite  combines  the  Arithmetic  Reasoning  (AR)  test  and  Verbal  (VE)  composite.  The 
Electronics  composite  combines  the  General  Science  (GS),  Arithmetic  Reasoning  (AR),  Math  Knowledge  (MK), 
and  Electronics  Information  (El)  tests  (Segall,  2004). 
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branches  of  the  US  military  for  enlistment  qualifieation.  Eaeh  military  serviee  braneh  ereates  its 
own  set  of  eomposites  used  to  qualify  enlistees  into  training  speeialties.  The  US  Air  Foree  uses 
four  olassifieation  eomposites:  Meehanieal,  Administrative,  General,  and  Eleetronies.  The 
ASVAB  has  been  validated  for  training  (Ree  &  Earles,  1991;  Ree,  Carretta,  &  Doub,  1998/1999) 
and  job  performanee  (Ree,  Earles,  &  Teaehout,  1994). 

UAS/RPA  sensor  operator  training  criteria.  The  UAS/RPA  sensor  operator  (lUOXl) 
training  path  begins  with  the  Airerew  Fundamentals  eourse  at  Joint  Base  San  Antonio  (JBSA)- 
Laekland,  TX.  The  eourse  is  6  days  long  and  prepares  enlisted  personnel  for  their  transition  to  a 
eareer  in  aviation.  This  eourse  sereens  for  the  ability  to  handle  the  rigor  of  airerew  duties  prior 
to  eandidates  entering  expensive  follow-on  training  (United  States  Air  Foree,  2012).  Next, 
students  eomplete  the  RPA  Basie  Sensor  Operator  Course  (BSOC)  at  JBSA-Randolph,  TX.  This 
6  week  eourse  provides  instruetion  to  students  in  areas  sueh  as  RPA  erew  duties,  types  of 
sensors,  exposure  to  weapons,  and  geospatial  referenee  systems.  Following  BSOC,  sensor 
operators  attend  MQ-1  or  MQ-9  Formal  Training  loeated  at  Holloman  AFB,  NM,  Mareh  ARB, 
CA,  or  Haneoek  Field  in  Syraeuse,  NY.  At  the  Formal  Training  Units,  sensor  operators  undergo 
the  MQ-1  or  MQ-9  Initial  Qualifieation  Course  lasting  approximately  three  to  four  months.  The 
MQ-1  and  MQ-9  eourses  are  designed  to  produee  basie  mission  eapable  airerew  in  MQ-1  and 
MQ-9  operations,  respeetively,  and  provide  training  in  areas  sueh  as  intelligenee,  surveillanee, 
and  reeonnaissanee  (ISR),  elose  air  support  (CAS),  and  eombat  seareh  and  reseue  (CSAR).  The 
MQ-1  and  MQ-9  eourses  also  are  used  to  provide  requalifieation  or  transition  training  for  MQ-1 
or  MQ-9  sensor  operators  that  have  been  unqualified  for  over  eight  years.  Graduates  from  the 
Formal  Training  Units  move  to  their  eombat  squadrons,  go  through  a  eombat  mission-ready 
eertifieation,  and  then  beeome  line  flyers. 
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Training  performance  criteria  were  collected  for  three  courses:  Basic  Sensor  Operator 
Course  (BSOC,  n  =  461),  MQ-IB  Initial  Qualification  and  Requalification  Training  (MQ-1 
IQRT,  n  =  430)  or  MQ-9  Initial  Qualification  and  Requalification  Training  (MQ-9  IQRT,  n  = 
249)  courses.  The  training  criterion  for  the  BSOC  course  was  the  average  grade  for  all  written 
tests  across  all  training  blocks.  The  criterion  for  the  MQ-1  IQRT  and  MQ-9  IQRT  courses  was  a 
4-point  final  grade  coded  as  (1)  fail,  (2)  pass,  (3)  satisfactory,  or  (4)  outstanding. 

Analyses 

Analyses  were  conducted  by  training  course.  They  began  with  an  examination  of  the 
descriptive  statistics  for  the  selection  test  scores  and  training  grades.  Three  sets  of  correlations 
were  examined:  observed  (uncorrected)  correlations,  correlations  corrected  for  multivariate  range 
restriction  (Eawley,  1943),  and  correlations  corrected  for  both  range  restriction  and  reliability 
(Hunter  &  Schmidt,  2004)  of  the  criterion.  The  assumptions  underlying  range  restriction 
correction  are  the  same  as  two  of  the  three  assumptions  underlying  the  computation  of  a  Pearson 
product-moment  correlation  -  linearity  of  form  and  homoscedasticity.  If  the  assumptions  are  met 
to  estimate  the  correlation  coefficient,  they  also  are  met  to  compute  the  correction.  The 
corrected  means,  standard  deviations,  and  correlations  are  superior  estimates  of  the  population 
values  compared  to  the  uncorrected  values.  This  method  removes  the  bias  from  the  uncorrected 
sample  estimates. 


The  range-restriction-corrected  correlations  were  then  corrected  for  reliability  of  the 


training  grades  =  —j=)-  The  reliability  of  the  pilot  training  grades  was  estimated  to  be  .80 


based  on  results  from  similar  studies  that  examined  academic  grades  (Kuncel,  Hezlett,  &  Ones, 


2004).  The  correlations  were  not  corrected  for  the  reliability  of  both  the  test  scores  and  criterion 


because  we  were  interested  in  the  specific  predictors  (ASVAB  scores)  and  not  the  theoretical 
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constructs  underlying  them  (cognitive  ability).  This  third  set  of  eorrelations  provides  a 
theoretical  estimate  of  the  validities  of  the  predictors  when  a  perfeetly  reliable  eriterion  is 
available. 

Finally,  stepwise  regression  analyses  were  eonducted  to  identify  the  best  combination  of 
ASVAB  tests  for  predieting  training  grades  for  each  course.  After  the  best  combination  of 
ASVAB  tests  had  been  identified,  the  data  were  corrected  for  range  restriction  and  reliability  of 
the  training  grades.  Cross-validation  shrinkage  was  estimated  for  these  models  using  a  non- 
sampling  approaeh  (Stein,  1960).  Stem’s  (1960)  equation  has  the  advantage  of  providing  an 
estimate  of  average  eross-validity  based  on  the  largest  available  sample. 

Results 

Means  and  SDs 

Table  1  summarizes  the  means  and  SDs  for  the  test  seores  and  training  grades  by  course. 
Similar  results  were  observed  for  each  course.  The  mean  ASVAB  scores  were  range-restrieted 
eompared  to  the  normative  values  where  the  means  and  SDs  are  50  and  10  for  the  tests  and  50 
and  28.29  for  the  composites.  For  the  General  (G)  and  Eleetronies  (E)  composites  whieh  are 
used  to  qualify  applieants  for  SO  training,  the  means  across  the  three  eourses  were  on  average 
0.77  (G)  and  0.89  (E)  SDs  above  the  normative  values  and  the  varianees  were  on  average  32.2% 
(G)  and  24.2%  (E)  of  the  normative  values. 

[Insert  Table  1  about  here] 

Correlations 

Table  2  summarizes  the  correlational  analyses.  For  the  ASVAB  composites,  14  of  15 
observed  validities  were  statistieally  significant  for  the  three  training  grades.  Cohen  (1988) 
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characterizes  eorrelations  of  .10  as  small,  .30  as  medium,  and  .50  or  greater  as  large.  The 
observed  validities  for  the  General  (G)  and  Eleetronies  (E)  composites  were  small  for  all  three 
eourses.  After  eorreetion  for  range  restrietion  and  reliability  of  the  training  grades,  the  validities 
for  the  G  and  E  eomposites  were  in  the  moderate  to  strong  range  (Cohen,  1988)  for  all  three 
eourses.  They  were  .541  and  .535  for  the  BSOC,  .583  and  .553  for  the  MQ-1  IQRT,  and  .357 
and  .334  for  the  MQ-9  IQRT  eourses.  On  the  test  level,  24  of  27  observed  validities  were 
statistieally  signifieant.  As  with  the  eomposite  seores,  the  observed  validities  for  the  tests  were 
small,  but  were  moderate  or  large  after  eorreetion  for  range  restrietion  and  reliability  of  the 
training  grades. 

Regression  Analyses 

Stepwise  regression  analyses  were  eondueted  to  determine  the  best  eombination  of 
ASVAB  tests  for  predieting  training  grades  for  eaeh  eourse.  These  analyses  identified  different 
best-fitting  regression  models  for  the  three  courses.  For  the  BSOC  eourse,  the  best-fitting  model 
included  GS,  AR,  AS,  and  VE  with  R  =  .366.  After  eorreetion  for  range  restrietion,  the  R 
inereased  to  .543,  and  after  eorreetion  for  both  range  restrietion  and  reliability  of  the  training 
grades  it  inereased  to  .607.  The  eross-validity  of  the  fully-eorreeted  model  was  estimated  (Stein, 
1960)  to  be  .596.  The  estimated  eross-validities  for  the  ASVAB  G  and  E  eomposites  were  .536 
and  .530. 

The  stepwise  regression  for  the  MQ-1  IQRT  eourse  yielded  a  model  with  two  test  seores, 
MK  and  VE,  with  R  =  .236^.  The  R  inereased  to  .562  after  the  eorrelations  were  eorrected  for 
range  restrietion  and  to  .628  after  correction  for  both  range  restrietion  and  reliability  of  the 
eriterion.  The  estimated  eross-validities  were  .596  for  the  test  score  model,  .579  for  G,  and  .548 

^  The  MK  and  VE  scores  are  the  components  of  the  USAF  Administrative  composite.  The  increment  in  predictive 
validity  for  the  stepwise  regression  model  compared  to  the  Administrative  composite  occurs  because  weights  for 
these  scores  are  optimized  in  the  stepwise  regression  model  for  predicting  the  MQ-1  IQR  training  grade. 
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for  E.  Eor  the  MQ-9  IQRT  criterion,  the  stepwise  regression  model  included  only  one  ASVAB 
test,  AS,  which  yielded  a  correlation  of  .192.  The  correlation  increased  to  .268  after  correction 
for  range  restriction  and  to  .333  after  correction  for  both  range  restriction  and  unreliability  of  the 
training  grades.  The  cross-validities  were  .314  for  AS,  .341  for  G,  and  .337  for  E. 

The  stepwise  models  produced  higher  correlations  for  the  training  grades  than  did  the 
ASVAB  G  and  E  composites  for  the  BSOC  and  MQ-1  IQRT  courses,  but  not  for  the  MQ-9 
IQRT  course.  The  small  differences  in  predictive  validity  for  the  best-fitting  test  score  models 
and  ASVAB  composites  and  the  lack  of  a  common  best- fitting  model  for  the  three  courses 
suggests  that  the  current  composites  should  be  retained  for  now. 

Discussion 

Consistent  with  prior  ASVAB  validation  studies  involving  other  Air  Force  career  fields 
(Ree  &  Earles,  1991;  Ree  et  ah,  1994,  1998/1999),  test  scores  were  predictive  of  training 
performance  for  sensor  operator  training.  After  correction,  validities  were  in  the  moderate  to 
large  range  (Cohen,  1988)  for  the  ASVAB  composites  and  tests. 

Both  observed  and  corrected  validities  tended  to  be  lower  for  the  MQ-9  IQRT  course 
compared  with  the  other  two  courses.  The  reason  for  this  is  unknown,  but  may  in  part  be 
because  the  MQ-9  IQRT  course  occurs  later  in  the  training  sequence  where  the  impact  of 
cognitive  ability  and  technical  knowledge  measured  by  the  ASVAB  are  mitigated  by  the  effects 
of  training  (weaker  students  have  been  eliminated,  acquisition  of  job-specific  knowledge  and 
skills). 

Results  of  UAS/RPA  job/task  analyses  (Bailey,  2008;  Crumley  &  Bailey,  1979; 
Chappelle  et  ah,  2010,  201 1;  Howse,  2011;  Paullin  et  ah,  201 1)  have  identified  several  critical 
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sensor  operator  SAOCs  not  adequately  measured  by  the  ASVAB.  Critical  skills  and  abilities  not 
measured  by  the  ASVAB  include  logic,  perceptual  and  psychomotor,  spatial  and  symbolic 
reasoning,  situation  awareness,  verbal  communication,  and  working  memory.  Other  critical 
characteristics  not  measured  by  the  ASVAB  include  work  environment  preferences  associated 
with  sensor  operator  tasks  (e.g.,  preferences  for  excitement,  uncertainty,  and  unpredictability  of 
the  task)  and  personality  traits  (e.g.,  patience).  Several  measures  have  been  developed  to  address 
these  gaps  in  measurement.  Measures  of  non-verbal  reasoning  and  working  memory  have  been 
developed  as  potential  additions  to  the  ASVAB;  however,  to  date,  data  collection  and 
psychometric  studies  have  not  been  completed.  The  Air  Force  Personnel  Center  (AFPC)  has 
begun  data  collection  for  sensor  operator  trainees  on  the  Test  of  Basic  Aviation  Skills  (TBAS, 
Carretta,  2005)  which  includes  measures  of  psychomotor  and  spatial  ability.  AFPC  also  has 
completed  initial  development  of  measures  of  person-environment  fit  and  multitasking  which 
may  have  applicability  for  UAS/RPA  sensor  operators.  Finally,  data  collection  has  begun  for 
two  personality  tests,  the  Tailored  Adaptive  Personality  Assessment  System  (TAP AS;  Knapp, 
Heffner,  &  White,  (2011)  and  the  Self-Description  Inventory  (SDI+,  Manley,  2011).  The  utility 
of  these  measures  will  be  examined  in  future  studies  as  data  become  available. 

Potential  Changes  in  Job  and  SAOC  Requirements  as  Technology  Matures 

As  technology  matures  it  is  expected  that  the  roles  played  by  UAS/RPA  pilots  and  sensor 
operators  will  change.  UAS/RPA  will  become  more  automated  and  autonomous,  requiring  less 
emphasis  on  active  control  (hands-on  flying  and  maneuvering  of  sensors)  and  more  on 
supervisory  control  and  operator-machine  teaming.  For  example.  General  Atomics  Aeronautical 
(2014)  recently  demonstrated  an  advanced  cockpit  ground  control  station  with  improved 
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graphics  and  video  designed  to  enable  interoperability  and  eompatibility  aeross  ElAS/RPA 
systems.  Inereased  automation  and  autonomy  may  enable  eombination  of  the  pilot  and  sensor 
operator  roles  and  perhaps  allow  a  single  operator  to  exert  supervisory  eontrol  over  multiple 
systems.  In  sueh  a  seenario,  eognitive  ability  and  job  knowledge  likely  will  remain  important, 
with  less  emphasis  on  psyehomotor  ability.  Further,  the  ideal  personality  profile  for  someone 
exerting  supervisory  control  as  opposed  to  aetive  control  may  be  different  and  should  be 
examined  (King,  2000). 

Implieations  of  teehnology  advances  should  be  monitored  elosely,  along  with  ehanges  in  the  core 
duties  of  the  SO  role  and  an  understanding  of  the  attributes  required  to  suoeessfully  perform 
them. 
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Table  1.  Means  and  SDs  for  Selection  Test  Scores  and  Training  Grades  by  Course 


Score 

Abbrev. 

BSOC 

MQ-1  IQRT 

MQ-9  IQRT 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Armed  Eorees 

AEQT 

72.00 

14.70 

74.69 

14.11 

72.50 

14.25 

Qualifieation  Test 

Mechanical 

M 

68.99 

19.03 

70.88 

19.41 

68.12 

19.14 

Administrative 

A 

73.42 

14.05 

75.86 

13.39 

74.08 

13.46 

General 

G 

70.87 

16.35 

73.43 

15.78 

71.01 

16.38 

Eleetronies 

E 

75.97 

14.22 

73.43 

14.21 

76.55 

13.56 

General  Scienee 

GS 

56.36 

6.15 

56.86 

6.33 

56.41 

6.36 

Arithmetie 

AR 

56.78 

6.05 

57.31 

6.03 

56.80 

5.75 

Reasoning 

Word  Knowledge 

WK 

53.79 

5.87 

55.00 

6.14 

53.93 

6.21 

Paragraph 

PC 

56.01 

5.18 

56.68 

5.40 

55.97 

5.17 

Comprehension 

Math  Knowledge 

MK 

58.29 

5.14 

58.76 

5.10 

58.68 

5.01 

Eleetronies 

El 

56.07 

7.82 

56.86 

8.11 

56.57 

7.89 

Information 

Auto  &  Shop 

AS 

51.73 

8.51 

52.11 

8.77 

51.18 

8.40 

Information 

Mechanical 

MC 

57.85 

7.55 

57.38 

7.60 

57.26 

5.81 

Comprehension 

Assembling 

AO 

59.00 

6.16 

58.34 

6.37 

58.71 

6.14 
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Objects 

Avg.  Training  95.91  2.77  3.16  0.54  2.95  0.75 

Grade 


Note.  The  sample  sizes  were;  BSCC  (461),  MQ-1  IQRT  (430),  and  MQ-9  (249).  Not  all 
participants  were  administered  a  version  of  the  ASVASB  that  included  the  AO  subtest.  The 
sample  sizes  for  AO  were:  BSOC  (107),  MQl-IQRT  (391),  and  MQ9-IQRT  (220). 
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Table  2,  Observed  and  Corrected  Correlations  between  ASVAB  Scores  and  Training  Grades 


Score 

BSOC  (n  = 

461) 

MQ-1  IQRT(n  = 

430) 

MQ-9  IQRT  (n 

=  249) 

r 

Tc 

rfc 

r 

rc 

rfc 

r 

rc 

rfc 

AFQT 

.245" 

.473 

.528 

.218" 

.544 

.608 

.154" 

.302 

.337 

M 

.336" 

.525 

.586 

.195" 

.499 

.557 

.223" 

.348 

.389 

A 

.207" 

.451 

.504 

.231" 

.549 

.613 

.114 

.27S 

.310 

G 

.261" 

.484 

.541 

OO 

o 

.522 

.583 

.179” 

.320 

.357 

E 

.244" 

.479 

.535 

o 

OO 

OO 

.495 

.553 

.160" 

.299 

.334 

GS 

.119’’ 

.390 

.436 

.178" 

.481 

.537 

.106 

.265 

.296 

AR 

24T 

.473 

.528 

.118^’ 

.436 

.487 

.135" 

.281 

.314 

WK 

.193" 

.413 

.461 

OO 

o 

.522 

.583 

.169” 

.315 

.352 

PC 

o 

(N 

OO 

.429 

.479 

.203" 

.526 

.588 

.125" 

.285 

.318 

MK 

.135” 

.393 

.439 

.151” 

.456 

.509 

.009 

.196 

.219 

El 

.207" 

.422 

.471 

.097" 

.341 

.381 

.155" 

.283 

.316 

AS 

.282c 

.408 

.456 

.118" 

.242 

.270 

.192” 

.268 

.333 

MC 

.258" 

.443 

.495 

.162" 

.368 

.411 

.161" 

.263 

.294 

AO 

.082 

.445 

.494 

o 

OO 

.555 

.620 

.117 

.322 

.360 

Notes.  The  eolumns  labeled  “r”  are  observed  eorrelations,  while  those  labeled  ‘Vc”  were 
corrected  for  range  restriction,  and  those  labeled  “rfc”  were  corrected  for  both  range  restriction 
and  reliability  of  the  training  grades.  Not  all  participants  were  administered  a  version  of  the 
ASVASB  that  included  the  AO  subtest.  The  sample  sizes  for  the  AO  score  were  107  for  BSOC, 
391  for  MQ-1  IQRT,  and  220  for  MQ-9  IQRT. 

"p<.05;'’p<.01;‘=p<.001 
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